First-in first-out function for segmented data stream processing

ABSTRACT

A method of segmented media data processing can include receiving a first sequence of first segments partitioned from a first data stream of a streaming media, and storing the first segments into a first first-in first-out (FIFO) buffer. In the first FIFO buffer, each first segment and attributes associated with each first segment form an entry of the first FIFO buffer. The attributes associated with each first segment can include a start time of the respective first segment, a duration of the respective first segment, and a length of the respective first segment indicating a number of bytes in the respective first segment. The first segments received from the first FIFO buffer can be processed using a first media processing task of a workflow in a network-based media processing (NBMP) system. The first segments received from the first FIFO buffer can be processed independently from each other.

INCORPORATION BY REFERENCE

This present disclosure claims the benefit of priority to U.S.Provisional Application No. 62/907,625, “Indexed FIFO Functions forSegmented Processing in Cloud” filed on Sep. 28, 2019, which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to first-in first-out (FIFO) buffers forsegmented data stream processing on a computing platform (e.g., acomputer or a cloud platform). Some embodiments are related withNetwork-based Media Processing (NBMP).

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent the work is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

Multimedia service providers face the challenge of adapting theirservices to multiple cloud and network service providers to reach theircustomers. These cloud and network service providers oftentimes definetheir own application programing interfaces (APIs) to assign computeresources to their customers. The Network-based Media Processing (NBMP)standard is being developed to address fragmentation and offer a unifiedway to perform media processing on top of any cloud platform and on anyIP network. NBMP defines a Workflow Manager that can use implementationsof ready-made media processing functions and compose them together tocreate a media processing workflow.

SUMMARY

Aspects of the disclosure provide a method of segmented media dataprocessing. The method can include receiving a first sequence of firstsegments partitioned from a first data stream of a streaming media, andstoring the first segments into a first first-in first-out (FIFO)buffer. In the first FIFO buffer, each first segment and attributesassociated with each first segment form an entry of the first FIFObuffer. The attributes associated with each first segment can include astart time of the respective first segment, a duration of the respectivefirst segment, and a length of the respective first segment indicating anumber of bytes in the respective first segment. The first segmentsreceived from the first FIFO buffer can be processed using a first mediaprocessing task of a workflow in a network-based media processing (NBMP)system. The first segments received from the first FIFO buffer can beprocessed independently from each other.

In an embodiment, the first sequence of the first segments are receivedfrom a continuous-to-segmented conversion function that partitions thefirst data stream of the streaming media from a continuous stream to asegmented stream. In an embodiment, the first sequence of the firstsegments partitioned from the first data stream of the streaming mediaare received from a media processing task of the workflow in the NBMPsystem. In an embodiment, the first data stream of the streaming mediais received from a NBMP source of the NBMP system, or from a mediaprocessing task of the workflow in the NBMP system.

An embodiment of the method can further includes storing the processedfirst segments output from the first media processing task of theworkflow into a second FIFO buffer. In the second FIFO buffer, eachprocessed first segment and attributes associated with each processedfirst segment form an entry of the second FIFO buffer. The attributesassociated with each processed first segment can include the start timeof the respective first segment, the duration of the respective firstsegment, and a length of the respective processed first segmentindicating a number of bytes in the respective processed first segment.In an embodiment, the entries stored in the second FIFO buffer areconverted to a continuous stream without the attributes associated witheach processed first segment.

An embodiment of the method can further include receiving a secondsequence of second segments partitioned from a second data stream of thestreaming media, and storing the second segments into a third FIFObuffer. In the third FIFO buffer, each second segment and attributesassociated with each second segment form an entry of the third FIFObuffer. The attributes associated with each second segment can include astart time of the respective second segment, a duration of therespective second segment, and a length of the respective second segmentindicating a number of bytes in the respective second segment. Thesecond segments received from the third FIFO buffer can be processedusing a second media processing task of the workflow in the NBMP system.The second segments received from the third FIFO buffer are processedindependently from each other.

Partitioning the first sequence of the first segments from the firstdata stream is synchronized with partitioning the second sequence of thesecond segments from the second data stream such that each first segmentof the first data stream and the respective second segment of the seconddata stream correspond to a same time span of the streaming media. In anembodiment, each first segment of the first data stream and therespective second segment of the second data stream have a same starttime in the respective attribute associated with each first or secondsegment.

Aspects of the disclosure provide an apparatus. The apparatus caninclude circuitry configured to receive a first sequence of firstsegments partitioned from a first data stream of a streaming media, andstore the first segments into a first FIFO buffer. In the first FIFObuffer, each first segment and attributes associated with each firstsegment form an entry of the first FIFO buffer. The attributesassociated with each first segment can include a start time of therespective first segment, a duration of the respective first segment,and a length of the respective first segment indicating a number ofbytes in the respective first segment. The first segments received fromthe first FIFO buffer can be processed using a first media processingtask of a workflow in a NBMP system. The first segments received fromthe first FIFO buffer are processed independently from each other.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, the nature, and various advantages of the disclosedsubject matter will be more apparent from the following detaileddescription and the accompanying drawings in which:

FIG. 1 shows an exemplary data processing system according to anembodiment of the disclosure.

FIG. 2 shows an example of a workflow according to an embodiment of thedisclosure.

FIG. 3 shows an example first-in first-out (FIFO) buffer according to anembodiment of the disclosure.

FIG. 4 shows an example continuous-to-segmented (C2S) conversionfunction according to an embodiment of the disclosure.

FIG. 5 shows an example segmented-to-continuous (S2C) conversionfunction according to an embodiment of the disclosure.

FIG. 6 shows an example processing path according to an embodiment ofthe disclosure.

FIG. 7 shows a multi-indexed segment FIFO (MiFIFO) function according toan embodiment of the disclosure.

FIG. 8 shows an example process of a segmented data stream processingprocess according to an embodiment of the disclosure.

FIG. 9 shows a schematic illustration of a computer system according toan embodiment of the disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

I. Network-Based Media Processing

Aspects of the disclosure describe segmented data stream processingmechanisms related with a network-based media processing (NBMP)framework. The NBMP framework can be employed to initialize and controlmedia processing in a network (e.g., on a cloud platform). For example,a NBMP source describes requested media processing and providesinformation about the nature and format of media data. In response, anNBMP workflow manager can establish a media processing workflow andinform the NBMP source that the workflow is ready, and the requestedmedia processing can start. The media source(s) can then starttransmitting media to the network for processing.

An NBMP workflow can be understood as a connected graph of mediaprocessing tasks, each of which performs a well-defined media processingoperation. The workflow manager ensures correct operation of theworkflow by configuring and monitoring each task as well as a workflowoutput. The workflow manager is responsible for selection of mediaprocessing functions and instantiating the selected functions as tasksbased on a workflow description that is received from the NBMP source.

By employment of techniques disclosed herein, a continuous data streamcan be converted to a sequence of segments. The segments can then bestored in a first-in first-out (FIFO) buffer with each segmentassociated with a set of attributes such as a start time, a duration,and a length of the respective segment. Such segments can be processedindependently through one or more tasks in a workflow. For example, thesegments can be processed in parallel using multiple instances of a samefunction. Stateless containers in a cloud computing plat form can thusbe applied to such segmented media stream processing.

FIG. 1 shows an exemplary NBMP system (100) according to an embodimentof the disclosure. The NBMP system (100) can support existing mediadelivery methods such as streaming, file delivery, push-basedprogressive download, hybrid delivery, multipath, and heterogeneousnetwork environments. The NBMP system (100) can also enable emergingmedia applications by providing various cloud-based media processingcapabilities such as network-assisted media quality enhancement (e.g.,virtual reality (VR) stitching, video up-scaling, mobile edge encodingfor adaptive streaming, and content-aware cloud transcoding),network-assisted media distribution (e.g., live media ingestion, onlinetranscoding, and media-aware caching), network-assisted mediacomposition (e.g., augmented video streaming, and customized mediacomposition), immersive media handling (e.g., network aggregated pointcloud media), and the like.

As shown, the NBMP system (100) can include a NBMP source (101), a NBMPworkflow manager (103), a function repository (105), a media source(111), a data processing entity (MPE) (113), and a media sink (115). TheNBMP system (100) can include additional media source(s), media sink(s),and/or MPEs.

The NBMP source (101) can describe, or otherwise indicate, mediaprocessing operations to be performed by the media processing entity(113). The function repository (105) can store various media processingfunctions. The NBMP source (101) and the NBMP workflow manager (103) canretrieve media processing functions from the function repository (105).A media processing function or a function can be used for performing aspecific media processing process, such as a stitching process in a 360video streaming service, an artificial intelligence (AI) based videoup-scaling process in a video delivery service, or the like.

A media processing task or a task can be a runtime instance of afunction that is executed by the MPE (113). An NBMP workflow or aworkflow can be represented by a graph (e.g., a directed acyclic graph(DAG)) of one or more connected task(s) that achieve the requested mediaprocessing. The NBMP workflow manager (103) or the workflow manager(103) can provision task(s) and connect the task(s) to create, control,manage and monitor a workflow, for example, based on a workflowdescription document (WDD).

The media source (111) can provide media content (e.g., media data,supplementary information) to be processed by a workflow. Thesupplementary information can include metadata or auxiliary informationrelated to the media data. The media source (111) can provide an inputto the workflow. The media sink (115) can consume an output of theworkflow. The MPE (113) can run one or more media processing task(s) toprocess the media content.

Different entities (e.g., the NBMP Source (101), the workflow manager(103) and the MPE (113)) in the NBMP system (100) can use APIs to invokeand respond to media service requests as shown in FIG. 1. The APIs caninclude a NBMP workflow API (or a workflow API), a function discoveryAPI, and a task API. The workflow API can provide an interface betweenthe NBMP Source (101) and the workflow manager (103). The task API canprovide an interface between the workflow manager (103) and mediaprocessing tasks. The function discovery API can provide an interfacebetween the workflow manager (103)/the NBMP Source (101) and theFunction Repository (105).

The NBMP interfaces described above can be used to create and controlmedia processing workflows in the network. The NBMP system (100) can besplit into a control plane and a media plane (or media data plane). Thecontrol plane can include the workflow API, the function discovery API,and the task API.

The workflow API can be used by the NBMP source (101) to create andcontrol a media processing workflow. The NBMP Source (101) can use theworkflow API to communicate with the workflow manager (103) forconfiguring and controlling media processing in the network. When theNBMP Source (101) sends a request to the workflow manager (103) byincluding a workflow resource (WR) in an operation of the workflow API,the workflow manager (103) can parse the WR, the included WDD andcorresponding descriptors, and take the appropriate actions according tothe requested operation. Then, the workflow manager (103) canacknowledge the request with a response. The workflow API operations caninclude creating a workflow (e.g., CreateWorkflow), updating a workflow(e.g., UpdateWorkflow), deleting a workflow (e.g., DeleteWorkflow),retrieving a workflow (e.g., RetrieveWorkflow), and the like.

The function discovery API can provide the means for the workflowmanager (103) and/or the NBMP Source (101) to discover media processingfunctions that can be loaded as part of a media processing workflow.

The task API can be used by the workflow manager (103) to configure andmonitor task(s) (e.g., a task 1 and a task 2 run by the MPE (113)) atruntime. The task API can define interface(s) for configuration of mediaprocessing tasks by the workflow manager (103), for example, after theresources for the task are allocated in the MPE (113). Task APIoperations can include creating a task (e.g., CreateTask), updating atask (e.g., UpdateTask), getting a task (e.g., GetTask), deleting a task(e.g., DeleteTask), and the like.

On the media plane, the media formats, the metadata, and thesupplementary information formats between the NBMP Source (111) andtask(s), as well as between the tasks can be defined.

A workflow description (WD) can be passed from the NBMP source (101) tothe workflow manager (103). The WD can describe information such asinput data and output data, functions and other requirements for theworkflow.

The workflow manager (103) can receive a WDD from the NBMP source (101)and can build a workflow for requested media processing. In a workflowprocedure, media processing functions can be selected, for example, fromthe function repository (105), and then corresponding media processingtasks can be configured and distributed to a set of one or more MPEs(e.g., including the MPE (113)).

The set of functions provided by the function repository (105) can beread by an NBMP source (101) and the workflow manager (103). In anembodiment, the NBMP source (101) requests the creation of a workflowusing a set of functions in the function repository (105). Accordingly,the NBMP source (101) is configured to select functions for theworkflow. The NBMP source (101) can request the creation of the workflowas described below. The NBMP source (101) can use a description of themedia processing tasks by which the workflow is to be created, and canspecify a connection map to define connections of inputs and outputs ofthe media processing tasks. When the workflow manager (103) receives theabove information from the NBMP source (101), the workflow manager (103)can instantiate the media processing tasks based on respective functionnames and can connect the media processing tasks according to theconnection map.

Alternatively, the NBMP source (101) can request the creation of aworkflow using a set of keywords by which the workflow manager (103) canconstruct the workflow. Accordingly, the NBMP source (101) may not beaware of a set of functions to be inserted into the workflow. The NBMPsource (101) can request the creation of the workflow as describedbelow. The NBMP source (101) can use the set of keywords by which theworkflow manager (103) can find the appropriate functions, and canspecify the requirements of the workflow using suitable workflowdescription.

When the workflow manager (103) receives the above information (e.g.,the set of keywords) from the NBMP source (101), the workflow manager(103) can create the workflow by searching for appropriate functionsusing the keywords, for example, specified in a processing descriptor.The workflow manager (103) can then use other descriptors in theworkflow description to provision the media processing tasks and connectthe media processing tasks to create the final workflow.

A processing model of the workflow manager (103) can be described asbelow.

The workflow manager (103) can discover available media processingfunctions as below. The NBMP function repository (105) can provide thefunction discovery interface (or API) to allow external entities toquery for a media processing function that can fulfil the requestedprocessing. The workflow manager (103) can have access to a directoryservice that offers a searchable list of media processing functions. Theworkflow manager (103) can use the description of the media processingtasks in the workflow description to find the appropriate functions forthe workflow.

Selection of the media processing tasks for the workflow can bedescribed below. When a request for media processing is received fromthe NBMP source (101), the workflow manager (103) can search thefunction repository (105) to find the list of all available functionsthat can fulfill the workflow. Using the workflow description from theNBMP Source (101), the workflow manager (103) can find the functionsfrom the function repository (105) to implement the workflow, which candepend on the information for media processing from the NBMP Source(101). The information for media processing can include the input andoutput description, the description of the requested processing, and theinformation in other descriptors for functions in the function directory(105). Mapping of the source requests to appropriate media processingtasks to be included in the workflow can be a part of the implementationof the NBMP in the network. To reference and link input sources withinput port names and output port names at the time of task creation, theinput-ports and output-ports can be used to make references to the inputstreams.

A search for appropriate functions to be instantiated as tasks can beperformed by the workflow manager (103) using a function discovery API.Alternatively, the workflow manager (103) can retrieve detailedinformation of some or all suitable functions in the function repository(105) using the function discovery API. The workflow manager (103) canthen compare the information for media processing from the NBMP source(101) with different descriptors of each function.

Selected media processing tasks can be configured in the workflow. Whenthe functions to be included in the workflow are identified, the NBMPworkflow manager (103) can instantiate the functions as respective tasksand configure the tasks so that the tasks can be added to the workflow.The NBMP workflow manager (103) can extract the configuration data fromthe media processing information received from the NBMP source (101) andconfigure the corresponding tasks. The configuration of the Tasks can beperformed using a task API (e.g., NBMP task API).

Task allocation and distribution can be described below. The workflowmanager (103) can use the workflow to perform processing deployment andconfigure the media processing entities. In an example, forcomputationally intensive media processing requests, the workflowmanager (103) can set up multiple computational instances and distributea workload among the multiple computational instances. Thus, theworkflow manager (103) can connect and configure the multiplecomputational instances as needed. In an example, the workflow manager(103) allocates a same task to multiple instances and provisions a loadbalancer to distribute the workload among the multiple instances using achosen scheduling mechanism. In an alternative example, the workflowmanager (103) allocates different operations of the same task todifferent instances (e.g., parallel operations). In both examplesdescribed above, the workflow manager (103) can set up the workflowpaths between the instances, and thus the suitable workload can besuccessfully realized. The workflow manager (103) can configure thetasks to push the processed media data/streams (or make them availablethrough a pull mechanism) to a next task in the workflow graph.

When the workflow manager (103) receives a WDD from the NBMP Source(101), the workflow manager (103) can perform a selection of mediaprocessing functions to be inserted into the workflow. When the list oftasks to be included in the workflow is compiled, the workflow manager(103) can then connect the tasks to prepare the workflow.

The workflow manager (103) can generate a workflow, for example, asrepresented by a graph (e.g., a DAG). FIG. 2 shows an example of a graph(e.g., a DAG) (200) according to an embodiment of the disclosure. TheDAG (200) can include a plurality of nodes (T1)-(T6) and a plurality oflinks (or connections) (202)-(208). In an example, the DAG (200)represents the workflow (200).

Each node of the DAG (200) can represent a media processing task in theworkflow (200). A link (e.g., the link (202)) connecting a first node(e.g., the node (T1)) to a second node (e.g., the node (T2)) in the DAG(200A) can represent a transfer of an output of the first node (e.g.,the node (T1)) as an input to the second node (e.g., the node (T2)).

In general, a workflow can include any suitable number of input(s) (orworkflow input(s)) and any suitable number of output(s) (or workflowoutput(s)). The workflow input(s) can be connected to the media source(111), other workflow(s), and/or the like, and the workflow output(s)can be connected to the media sink (115), other workflow(s), and/or thelike. The workflow (200) has an input (201) and outputs (209) and (210).The workflow (200) can have one or more outputs from intermediate nodesin some embodiments.

II. First-in First-Out (FIFO) Function for Segmented Data StreamProcessing

FIG. 3 shows an example FIFO buffer (300) according to an embodiment ofthe disclosure. The FIFO buffer (300) can be used for implementing aFIFO function for segmented data stream processing. The FIFO buffer(300) can receive a sequence of input segments (310) as an input, andstore the received segments as entries (301A-301N).

The sequence of segments (310) can be received from acontinuous-to-segmented (C2S) conversion function. The C2S function canreceive a data stream of a streaming media, for example, from the mediasource (111) in the FIG. 1 example, and partition the data stream intothe sequence of the segments (310). For example, the streaming media canbe a delivery of digital content (e.g., a television program or a movie)over the Internet by a streaming service provider. The data stream canbe a bitstream carrying one of various types of content of the streamingmedia. Examples of the data stream can include a video stream, an audiostream, a metadata stream, a point cloud stream, or the like.

Or, the sequence of segments (310) can be received from a task in aworkflow, for example, in the NBMP system (100). A data stream can bepartitioned into a sequence of segments before being fed into the taskfor processing. The task then outputs the sequence of segments (310).

The entries (310A-310N) can each include a segment belonging to thesequence of segments (310) and a set of attributes associated with thesegment. For example, the attributes of each segment can be associatedto the respective segment when the respective segment is generated(e.g., partitioned from a data stream), or after the respective isprocessed by a task.

In an embodiment, the attributes in each entry include a start timedenoted by S_(i) indicating a start time of the respective segment, aduration D_(i), indicating a duration of the respective segment whendisplayed, and a length L_(i) (or a size) indicating a length of therespective segment, for example, in terms of a number of bytes in therespective segment. The parameter i represents an index corresponding toeach segment. As shown, the entry (301A) includes attributes S₀, D₀, andL₀; the entry (301B) includes attributes S₁, D₁, and L₁; and the entry(301B) includes attributes S_(n-1), D_(n-1), and L_(n-1).

In an example, the start time of each segment in the sequence ofsegments (310) can be a time with respect to a start time of the datastream the segment is partitioned from, or any time within the span ofthe data stream designated as a time reference. Or, the start time canbe a time with respect to a timing reference, such as a clock providinga clock signal. The duration of each segment in the sequence of segments(310) can be a time span corresponding to a period of time fordisplaying the segment when the data stream of the streaming media isbeing displayed. The start time and the duration of each segment can berepresented in units of second, millisecond, or the like. In variousembodiments, the durations of each segment in the sequence of segments(310) can be the same or different.

In an embodiment, the set of attributes of each segment includes onlythe attribute of the start time, and the attributes of the duration andthe length are optional. For example, when the duration and the lengthfor each segment are equal, the attributes of the duration and thelength can be omitted. In an embodiment, the set of attributes of eachsegment includes an index indicating an order of each segment in thesequence of segments (310). In an example, when the index attribute isused, the attributes of the start time, the duration, and the length canbe optional. Both the start time attribute and the index attribute canserve the purpose of indexing each segment. Accordingly, the FIFO buffer(300) storing the indexed segments can be referred to as an indexedsegment FIFO denoted by iFIFO.

In an embodiment, each entry (301A-301N) can be read from the FIFObuffer (300) to provide a sequence of output segments (320) in afirst-in first-out manner. In an example, the sequence of input segments(310) have been processed by one or more of tasks of a workflow, andthen stored in the FIFO buffer (300). The output segments (320) readfrom the FIFO buffer (300) is then concatenated together by asegmented-to-continuous (S2C) conversion function to form a continuousdata stream. The continuous data stream can then be output from theworkflow.

In an example, the sequence of input segments (310) are partitioned froma data stream of a streaming media in a previous step. The outputsegments (320) read from the FIFO buffer (300) is then input to a taskof a workflow for processing. In an example, the sequence of inputsegments (310) have been processed by one or more of tasks of aworkflow, and then stored in the FIFO buffer (300). The output segments(320) read from the FIFO buffer (300) is then input to a next task ofthe workflow for further processing.

In an embodiment, the FIFO buffer (300) can be configured with a set ofconfiguration parameters. In an example, the configuration parameterscan include a size of the FIFO buffer (300), for example, in a number ofentries, and a maximum size of the entries in the FIFO buffer (300). Inan example, the configuration parameters can be received from the NBMPsource (101) in the FIG. 1 example that specifies how the FIFO buffer(300) is to be configured and operated in a workflow. In an example, theconfiguration parameter is determined by the NBMP workflow manager (103)based on information received from the NBMP source (101). For example,the information can include parameters describing a data stream of astreaming media to-be-processed with a workflow, and/or parametersdescribing a function that is before or after the FIFO buffer (300) inthe workflow. The NBMP workflow manager (103) can then determine how toconfigure the FIFO buffer (300) in the workflow based on thatinformation.

FIG. 4 shows an example C2S conversion function (400) according to anembodiment of the disclosure. The C2S conversion function (400) can beconfigured to receive a continuous data stream (410) of a streamingmedia, and partition the continuous data stream (410) into a segmenteddata stream (420).

The segmented data stream (420) can include a sequence of segments(421A-421N) each associated with a set of attributes, such as attributesof a start time, a duration, and a length of the respective segment. Thesegmented data stream 420 can then be input to a FIFO function, such asthe FIFO function (300).

The data stream (410) can be a data stream received from the mediasource (111), and is input to the C2S conversion function (400) beforeentering a workflow for processing. Or, the data stream (410) can be adata stream that has been processed by a function of a workflow beforereaching the C2S conversion function (400).

Partitioning operation of the C2S conversion function (400) can beperformed in such a way that the resulting segments (421A-421N) can besubsequently processed independently. In an example, the resultingsegments (421A-421N) can be processed in parallel in a workflow withmultiple instances of a same media processing function in the workflowto accelerate processing speed.

In an example, the resulting segments (421A-421N) can be processed witha step mode. For example, in a computing platform (e.g., a computer or acloud platform) with limited resources, tasks belonging to a sameworkflow may not be run at the same time. Instead, the tasks connectedin series can be instantiated successively to process a group ofsegments. A current group of segments can be processed step by step(task by task) passing through the workflow before a next group ofsegments can enter the workflow.

In an embodiment, containerized tasks are employed in a cloud computingenvironment where a task can run within a stateless container that runson top of a single operating system kernel and emulates an operatingsystem rather than the underlying hardware. The independent segments(421A-421N) resulting from the C2S conversion function (400) can thus besuitably processed within such containers either in parallel or in stepmode.

The partitioning operation at the C2S conversion function (400) candepend on internal structures of the data stream (410). In variousexamples, the data stream (410) may be of various types and have variousformats, and can be, for example, a point cloud data stream, a videostream, an audio stream, or the like. Each type of data stream may havea specific internal structure. The C2S conversion function (400) canidentify the different structures, and accordingly partition therespective data stream into segments. While partitioning, a set ofattributes (e.g., a start time, a duration, a length (size), or thelike) of the respective segment can be determined and associated withthe respective segment.

For example, the data stream (410) can be a video including structuresof group of pictures (GOPs). Pictures (or frames) within a GOP canreference other members of the GOP for inter prediction coding. However,each GOP in the video can be processed independently. Accordingly, thepartitioning can be performed on a basis of GOPs. For example, one ormore GOPs in the video can be included in one segment. The resultingsegments can thus be processed independently.

In another example, a video compliant with High Efficiency Video Coding(HEVC) standard may include structures of random access point (RAP)pictures to facilitate users jumping to specific parts of the video.Such RAP pictures (e.g., an instantaneous decoding refresh (IDR)picture) can be used as a start of a segment for the partitioningoperation.

In addition, while partitioning the video in the above examples of thepartitioning operation, a start time of each segment can be determinedbased on the location where a partition operation takes place in thevideo. For example, in the above examples, each picture can beassociated with a picture order count (POC) indicating an order ofpictures in the video. Based on the POC information, a timing of apicture where the partition takes place can be determined and associatedwith respect segment. Similarly, based on information of the internalstructure of the video, a duration and a length (size) of the respectivesegment can be determining while partitioning the video.

FIG. 5 shows an example S2C conversion function (500) according to anembodiment of the disclosure. The S2C conversion function (500) can beconfigured to receive a segmented data stream (510), and convert thesegmented date stream (510) to a continuous data stream (520). Thesegmented data stream (510) can include a sequence of segments(511A-511N) each associated with a set of attributes. For example, theset of attributes can include a start time, a duration, and a length ofthe respective segment. The sequence of segments (511A-511N) can be anoutput of a task in a workflow.

In an example, the S2C conversion function (500) removes the attributesassociated with the segments (511A-511N), and concatenates the sequenceof the segments (511A-511N) to generate the continuous data stream(520). The continuous data stream (520) can then be input to a next taskof the workflow for further processing. Or, the continuous data stream(520) can be transmitted to a media sink as an output of the workflow.

FIG. 6 shows an example processing path (600) according to an embodimentof the disclosure. The processing path can be part of a workflow in aNBMP system. The processing path (600) includes a series of a C2Sconversion function (601), a first iFIFO buffer (602), a task (603), asecond iFIFO buffer (604), and a S2C conversion function (605).

A continuous data stream (610) is received at the C2S conversionfunction (601). The continuous data stream (610) can be an output of aprevious task, or transmitted from a media source such as the mediasource (111) in the FIG. 1 example. The C2S conversion function (601)partitions the continuous stream (601) into a sequence of segments eachassociated with a set of attributes (e.g., a start time, a duration, alength, or the like). The sequence of segments is output from the C2Sconversion function (601) as a segmented data stream (620). The iFIFObuffer (602) receives the segmented data stream (620) and stores thesegments and associated attributes as entries.

The entries are then read from the iFIFO buffer (602) and processed bythe task (603). The task (603) can be a media processing taskinstantiated from a media processing function. In an example, the task(603) can be configured to be multiple instances of a same function.Accordingly, the segments in the iFIFO buffer (602) can be fed to themultiple instances and processed in parallel. In an example, the task(603) can be configured to operate in step mode. Specifically, the task(603) can be instantiated to process a group of segments received fromthe iFIFO buffer (602) and then terminated.

The iFIFO buffer (604) receives processed segments each associated withthe attributes from the task (603), and stores the processed segmentsand attributes in form of entries. The entries in the iFIFO buffer (604)are then input to the S2C conversion function (605) as a segmented datastream (630). The S2C conversion function (605) concatenates thesegments in the segmented data stream (630) to generate a continuousdata stream (640).

While only one task (603) is shown in the processing path (600), therecan be more than one tasks configured in place of the task (603) inother examples. Between a pair of consecutive tasks among those tasks,there can be an iFIFO buffer employed to connect the pair of tasks. Ofcourse, some pairs of consecutive tasks can be directly connected ineach pair without an iFIFO buffer.

FIG. 7 shows a multi-indexed segment FIFO (MiFIFO) function (700)according to an embodiment of the disclosure. The MiFIFO function (700)can include multiple FIFO buffers (730A-730N) operating in parallel. TheMiFIFO function (700) can receive multiple segment inputs (710A-710N)(e.g., 2, 3, or more segment inputs) at the multiple FIFO buffers(730A-730N), respectively. Each segment input (710A-710N) can be asequence of segments that are each associated with a set of attributes.The sequence of segments and the associated attributes are stored asentries in each FIFO buffer (730A-730N). For example, the FIFO buffer(730A) can include entries (701A-701N), the FIFO buffer (730B) caninclude entries (702A-702N), and the FIFO buffer (730N) can includeentries (703A-703N). The entries in each FIFO buffer (730A-730N) can beread out in a first-in first-out manner as multiple segment outputs(720A-720N).

In an example, data streams corresponding to the segment inputs(710A-710N) can belong to a same streaming media. For example, the datastreams can include one or more video streams, one or more audiostreams, one or more metadata streams, or the like. When beingdisplayed, contents belonging to different data streams butcorresponding to a same time span can be displayed synchronously. Toserve this purpose, contents belonging to different data streams butcorresponding to a same span are desired to be processed by a workflowin a synchronized manner.

Accordingly, in an embodiment, segmentation of the segments in eachsegment input (710A-710N) can be performed synchronously. For example,the locations in each data stream where the segmentation takes place canbe time aligned. As a result, the resulting segments overlapping a sametime span can be time aligned, and have a same start time and a sameduration. For example, the entries 701A, 702A, and 703A can have a samestart time and a same duration. During processing by one or morefunctions before or after the MiFIFO function (700) in the workflow,segments belonging to different data streams but corresponding to a sametime span can be processed in a synchronous way such that contents ofdifferent data stream but the same time span can be delivered to a usefor display properly.

In various embodiments, the segment inputs (710A-710N) can each beoutput from a different C2S conversion function, or a different mediaprocessing function. Or, two or more segments inputs (710A-710N) can beoutput from a same C2S conversion function or a same media processingfunction. Similarly, the segment outputs (720A-720N) can each be outputto a different S2C conversion function or a different media processingfunction. Or, two or more segment outputs (720A-720N) can be provided toa same S2C conversion function or a same media processing function.

In a workflow including multiple tasks and multiple processing paths,MiFIFO functions can be configured to each connect two groups of tasks:one group of tasks resides at an input side, and the other group oftasks resides at an output side of the respective MiFIFO function.

It is noted that the FIFO buffer mechanism disclosed herein is notlimited to a NBMP system although the iFIFO function and MiFIFO functionin some examples are explained in the context of NBMP. The FIFO bufferscan be applied to media data or non-media data processing scenarioswhere a workflow-like process including multiple tasks is employed.

In addition, the FIFO functions disclosed herein can be implemented byprocessing circuitry (e.g., one or more processors or one or moreintegrated circuits). Or, the FIFO functions disclosed herein can eachbe implemented as instructions of a program stored in a non-transitorycomputer-readable storage medium. The instructions, when executed by oneor more processors, can cause the processors to carry out a FIFOfunction.

II. Example Processes of Segmented Data Stream Processing

FIG. 8 shows an example process (800) of a segmented data streamprocessing process according to an embodiment of the disclosure. Theprocess (800) can be part of a process for media processing using aworkflow in a NBMP system. The process (800) can start from (S801) andproceed to (S810).

At (S810), a sequence of segments partitioned from a data stream of astreaming media can be received. For example, the sequence of segmentscan be received from a C2S conversion function or from a mediaprocessing task of the workflow in the NBMP system. The streaming mediacan include one or more data streams, and be transmitted from a mediasource over a network (e.g., the Internet).

At (S820), the segments can be stored into a first FIFO buffer. Forexample, each segment and associated one or more attributes can bestored as an entry in the first FIFO buffer. The attributes can includeone or more of a start time, a duration, and a length of the respectivesegments.

At (S830), the segments are read from the first FIFO buffer andprocessed by a task in the workflow. For example, the segments areprocessed independently from each other. In one example, the segmentsare processed in parallel. In an example, tasks in the workflow operatein step mode. Accordingly, the segments can be organized into groupsthat are each processed independently by the workflow and pass throughthe workflow step by step.

At (S840), after being processed at (S830), the segments can then bestored in a second FIFO buffer with associated attributes. For example,the attributes of each processed segments can have the same start timeand duration as the attributes associated with the segments before beingprocessed at (S830). Each processed segment can have a length attributeindicating a size (e.g., a number of bytes) of the processed segmentthat may have changed compared with the length before being processed.

At (S850), the processed segments can then be read from the second FIFObuffer and input to a S2C function, and be converted to a continuousdata stream. The continuous data stream can be output from the work flowin an example. The process (800) can then proceed to (S899) andterminate at (S899).

IV. Computer System

The methods, embodiments, entities, or functions disclosed herein may beimplemented by processing circuitry (e.g., one or more processors or oneor more integrated circuits). In one example, the one or more processorsexecute a program that is stored in a non-transitory computer-readablemedium to perform the methods, embodiments, or functions disclosedherein. The program can be coded using any suitable machine code orcomputer language, that may be subject to assembly, compilation,linking, or like mechanisms to create code comprising instructions thatcan be executed directly, or through interpretation, micro-codeexecution, and the like, by one or more CPUs, GPUs, and the like. Theinstructions can be executed on various types of computers or componentsthereof, including, for example, personal computers, tablet computers,servers, smartphones, gaming devices, internet of things devices, andthe like.

FIG. 9 shows a computer system (900) suitable for implementing certainembodiments of the disclosed subject matter. The components shown inFIG. 9 for computer system (900) are exemplary in nature and are notintended to suggest any limitation as to the scope of use orfunctionality of the computer software implementing embodiments of thepresent disclosure. Neither should the configuration of components beinterpreted as having any dependency or requirement relating to any oneor combination of components illustrated in the exemplary embodiment ofa computer system (900).

Computer system (900) may include certain human interface input devices.Such a human interface input device may be responsive to input by one ormore human users through, for example, tactile input (such as:keystrokes, swipes, data glove movements), audio input (such as: voice,clapping), visual input (such as: gestures), olfactory input (notdepicted). The human interface devices can also be used to capturecertain media not necessarily directly related to conscious input by ahuman, such as audio (such as: speech, music, ambient sound), images(such as: scanned images, photographic images obtain from a still imagecamera), video (such as two-dimensional video, three-dimensional videoincluding stereoscopic video).

Input human interface devices may include one or more of (only one ofeach depicted): keyboard (901), mouse (902), trackpad (903), touchscreen (910), data-glove (not shown), joystick (905), microphone (906),scanner (907), camera (908).

Computer system (900) may also include certain human interface outputdevices. Such human interface output devices may be stimulating thesenses of one or more human users through, for example, tactile output,sound, light, and smell/taste. Such human interface output devices mayinclude tactile output devices (for example tactile feedback by thetouch-screen (910), data-glove (not shown), or joystick (905), but therecan also be tactile feedback devices that do not serve as inputdevices), audio output devices (such as: speakers (909), headphones (notdepicted)), visual output devices (such as screens (910) to include CRTscreens, LCD screens, plasma screens, OLED screens, each with or withouttouch-screen input capability, each with or without tactile feedbackcapability-some of which may be capable to output two dimensional visualoutput or more than three dimensional output through means such asstereographic output; virtual-reality glasses (not depicted),holographic displays and smoke tanks (not depicted)), and printers (notdepicted).

Computer system (900) can also include human accessible storage devicesand their associated media such as optical media including CD/DVD ROM/RW(920) with CD/DVD or the like media (921), thumb-drive (922), removablehard drive or solid state drive (923), legacy magnetic media such astape and floppy disc (not depicted), specialized ROM/ASIC/PLD baseddevices such as security dongles (not depicted), and the like.

Those skilled in the art should also understand that term “computerreadable media” as used in connection with the presently disclosedsubject matter does not encompass transmission media, carrier waves, orother transitory signals.

Computer system (900) can also include an interface (954) to one or morecommunication networks (955). Networks can for example be wireless,wireline, optical. Networks can further be local, wide-area,metropolitan, vehicular and industrial, real-time, delay-tolerant, andso on. Examples of networks include local area networks such asEthernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G,LTE and the like, TV wireline or wireless wide area digital networks toinclude cable TV, satellite TV, and terrestrial broadcast TV, vehicularand industrial to include CANBus, and so forth. Certain networkscommonly require external network interface adaptors that attached tocertain general purpose data ports or peripheral buses (949) (such as,for example USB ports of the computer system (900)); others are commonlyintegrated into the core of the computer system (900) by attachment to asystem bus as described below (for example Ethernet interface into a PCcomputer system or cellular network interface into a smartphone computersystem). Using any of these networks, computer system (900) cancommunicate with other entities. Such communication can beuni-directional, receive only (for example, broadcast TV),uni-directional send-only (for example CANbus to certain CANbusdevices), or bi-directional, for example to other computer systems usinglocal or wide area digital networks. Certain protocols and protocolstacks can be used on each of those networks and network interfaces asdescribed herein.

Aforementioned human interface devices, human-accessible storagedevices, and network interfaces can be attached to a core (940) of thecomputer system (900).

The core (940) can include one or more Central Processing Units (CPU)(941), Graphics Processing Units (GPU) (942), specialized programmableprocessing units in the form of Field Programmable Gate Areas (FPGA)(943), hardware accelerators (944) for certain tasks, graphics adapters(950), and so forth. These devices, along with Read-only memory (ROM)(945), Random-access memory (946), internal mass storage such asinternal non-user accessible hard drives, SSDs, and the like (947), maybe connected through a system bus (948). In some computer systems, thesystem bus (948) can be accessible in the form of one or more physicalplugs to enable extensions by additional CPUs, GPU, and the like. Theperipheral devices can be attached either directly to the core's systembus (948), or through a peripheral bus (949). In an example, the screen(910) can be connected to the graphics adapter (950). Architectures fora peripheral bus include PCI, USB, and the like.

CPUs (941), GPUs (942), FPGAs (943), and accelerators (944) can executecertain instructions that, in combination, can make up theaforementioned computer code. That computer code can be stored in ROM(945) or RAM (946). Transitional data can be also be stored in RAM(946), whereas permanent data can be stored for example, in the internalmass storage (947). Fast storage and retrieve to any of the memorydevices can be enabled through the use of cache memory, that can beclosely associated with one or more CPU (941), GPU (942), mass storage(947), ROM (945), RAM (946), and the like.

The computer readable media can have computer code thereon forperforming various computer-implemented operations. The media andcomputer code can be those specially designed and constructed for thepurposes of the present disclosure, or they can be of the kind wellknown and available to those having skill in the computer software arts.

As an example and not by way of limitation, the computer system havingarchitecture (900), and specifically the core (940) can providefunctionality as a result of processor(s) (including CPUs, GPUs, FPGA,accelerators, and the like) executing software embodied in one or moretangible, computer-readable media. Such computer-readable media can bemedia associated with user-accessible mass storage as introduced above,as well as certain storage of the core (940) that are of non-transitorynature, such as core-internal mass storage (947) or ROM (945). Thesoftware implementing various embodiments of the present disclosure canbe stored in such devices and executed by core (940). Acomputer-readable medium can include one or more memory devices orchips, according to particular needs. The software can cause the core(940) and specifically the processors therein (including CPU, GPU, FPGA,and the like) to execute particular processes or particular parts ofparticular processes described above, including defining data structuresstored in RAM (946) and modifying such data structures according to theprocesses defined by the software. In addition or as an alternative, thecomputer system can provide functionality as a result of logic hardwiredor otherwise embodied in a circuit (for example: accelerator (944)),which can operate in place of or together with software to executeparticular processes or particular parts of particular processesdescribed herein. Reference to software can encompass logic, and viceversa, where appropriate. Reference to a computer-readable media canencompass a circuit (such as an integrated circuit (IC)) storingsoftware for execution, a circuit embodying logic for execution, orboth, where appropriate. The present disclosure encompasses any suitablecombination of hardware and software.

While this disclosure has described several exemplary embodiments, thereare alterations, permutations, and various substitute equivalents, whichfall within the scope of the disclosure. It will thus be appreciatedthat those skilled in the art will be able to devise numerous systemsand methods which, although not explicitly shown or described herein,embody the principles of the disclosure and are thus within the spiritand scope thereof.

What is claimed is:
 1. A method, comprising: receiving a first sequenceof first segments partitioned from a first data stream of a streamingmedia and receiving respective attributes associated with each of thereceived first segments; storing the first segments into a firstfirst-in first-out (FIFO) buffer, each first segment and the respectiveattributes form an entry of the first FIFO buffer, the attributesassociated with each first segment including a start time of therespective first segment, a duration of the respective first segment,and a length of the respective first segment indicating a number ofbytes in the respective first segment; and processing the first segmentsoutput from the first FIFO buffer using a first media processing task ofa workflow in a network-based media processing (NBMP) system, whereinthe first segments and the respective attributes are output from thefirst FIFO buffer entry by entry and processed independently from eachother.
 2. The method of claim 1, wherein the receiving the firstsequence of the first segments includes: receiving the first sequence ofthe first segments from a continuous-to-segmented conversion functionthat partitions the first data stream of the streaming media from acontinuous stream to a segmented stream.
 3. The method of claim 1,wherein the receiving the first sequence of the first segments includes:receiving the first sequence of the first segments partitioned from thefirst data stream of the streaming media from a second media processingtask of the workflow in the NBMP system.
 4. The method of claim 1,wherein the first data stream of the streaming media is received from aNBMP source of the NBMP system, or from a third media processing task ofthe workflow in the NBMP system.
 5. The method of claim 1, furthercomprising: storing the processed first segments output from the firstmedia processing task of the workflow into a second FIFO buffer in whicheach processed first segment and attributes associated with eachprocessed first segment form an entry of the second FIFO buffer, theattributes associated with each processed first segment including thestart time of the respective processed first segment, the duration ofthe respective processed first segment, and a length of the respectiveprocessed first segment indicating a number of bytes in the respectiveprocessed first segment.
 6. The method of claim 5, further comprising:converting the entries stored in the second FIFO buffer to a continuousstream without the attributes associated with each processed firstsegment.
 7. The method of claim 1, further comprising: receiving asecond sequence of second segments partitioned from a second data streamof the streaming media; storing the second segments into a third FIFObuffer in which each second segment and attributes associated with eachsecond segment form an entry of the third FIFO buffer, the attributesassociated with each second segment including a start time of therespective second segment, a duration of the respective second segment,and a length of the respective second segment indicating a number ofbytes in the respective second segment; and processing the secondsegments output from the third FIFO buffer using a fourth mediaprocessing task of the workflow in the NBMP system wherein the secondsegments output from the third FIFO buffer are processed independentlyfrom each other, wherein partitioning the first sequence of the firstsegments from the first data stream is synchronized with partitioningthe second sequence of the second segments from the second data streamsuch that each first segment of the first data stream and acorresponding synchronized second segment of the second data streamcorrespond to a same time span of the streaming media.
 8. The method ofclaim 7, wherein each first segment of the first data stream and thecorresponding synchronized second segment of the second data stream havea same start time in the respective attribute associated with each firstor second segment.
 9. An apparatus, comprising: circuitry configured to:receive a first sequence of first segments partitioned from a first datastream of a streaming media and receive respective attributes associatedwith each of the received first segments; store the first segments intoa first first-in first-out (FIFO) buffer, each first segment and therespective attributes form an entry of the first FIFO buffer, theattributes associated with each first segment including a start time ofthe respective first segment, a duration of the respective firstsegment, and a length of the respective first segment indicating anumber of bytes in the respective first segment; and process the firstsegments output from the first FIFO buffer using a first mediaprocessing task of a workflow in a network-based media processing (NBMP)system, wherein the first segments and the respective attributes areoutput from the first FIFO buffer entry by entry and processedindependently from each other.
 10. The apparatus of claim 9, wherein thecircuitry is further configured to: receive the first sequence of thefirst segments from a continuous-to-segmented conversion function thatpartitions the first data stream of the streaming media from acontinuous stream to a segmented stream.
 11. The apparatus of claim 9,wherein the circuitry is further configured to: receive the firstsequence of the first segments partitioned from the first data stream ofthe streaming media from a second media processing task of the workflowin the NBMP system.
 12. The apparatus of claim 9, wherein the first datastream of the streaming media is received from a NBMP source of the NBMPsystem, or from a third media processing task of the workflow in theNBMP system.
 13. The apparatus of claim 9, wherein the circuitry isfurther configured to: store the processed first segments output fromthe first media processing task of the workflow into a second FIFObuffer in which each processed first segment and attributes associatedwith each processed first segment form an entry of the second FIFObuffer, the attributes associated with each processed first segmentincluding the start time of the respective processed first segment, theduration of the respective processed first segment, and a length of therespective processed first segment indicating a number of bytes in therespective processed first segment.
 14. The apparatus of claim 13,wherein the circuitry is further configured to: convert the entriesstored in the second FIFO buffer to a continuous stream without theattributes associated with each processed first segment.
 15. Theapparatus of claim 9, wherein the circuitry is further configured to:receive a second sequence of second segments partitioned from a seconddata stream of the streaming media; store the second segments into athird FIFO buffer in which each second segment and attributes associatedwith each second segment form an entry of the third FIFO buffer, theattributes associated with each second segment including a start time ofthe respective second segment, a duration of the respective secondsegment, and a length of the respective second segment indicating anumber of bytes in the respective second segment; and process the secondsegments output from the third FIFO buffer using a fourth mediaprocessing task of the workflow in the NBMP system wherein the secondsegments output from the third FIFO buffer are processed independentlyfrom each other, wherein partitioning the first sequence of the firstsegments from the first data stream is synchronized with partitioningthe second sequence of the second segments from the second data streamsuch that each first segment of the first data stream and acorresponding synchronized second segment of the second data streamcorrespond to a same time span of the streaming media.
 16. The apparatusof claim 15, wherein each first segment of the first data stream and thecorresponding synchronized second segment of the second data stream havea same start time in the respective attribute associated with each firstor second segment.
 17. A non-transitory computer-readable medium storinginstructions that, when executed by a processor, cause the processor toperform a method, the method comprising: receiving a first sequence offirst segments partitioned from a first data stream of a streaming mediaand receiving respective attributes associated with each of the receivedfirst segments; storing the first segments into a first first-infirst-out (FIFO) buffer, each first segment and the respectiveattributes form an entry of the first FIFO buffer, the attributesassociated with each first segment including a start time of therespective first segment, a duration of the respective first segment,and a length of the respective first segment indicating a number ofbytes in the respective first segment; and processing the first segmentsoutput from the first FIFO buffer using a first media processing task ofa workflow in a network-based media processing (NBMP) system, whereinthe first segments and the respective attributes are output from thefirst FIFO buffer entry by entry and processed independently from eachother.
 18. The non-transitory computer-readable medium of claim 17,wherein the receiving the first sequence of the first segments includes:receiving the first sequence of the first segments from acontinuous-to-segmented conversion function that partitions the firstdata stream of the streaming media from a continuous stream to asegmented stream.
 19. The non-transitory computer-readable medium ofclaim 17, wherein the receiving the first sequence of the first segmentsincludes: receiving the first sequence of the first segments partitionedfrom the first data stream of the streaming media from a second mediaprocessing task of the workflow in the NBMP system.
 20. Thenon-transitory computer-readable medium of claim 17, wherein the firstdata stream of the streaming media is received from a NBMP source of theNBMP system, or from a third media processing task of the workflow inthe NBMP system.